Inward/outward vehicle monitoring for remote reporting and in-cab warning enhancements

ABSTRACT

Systems and methods are provided for intelligent driving monitoring systems, advanced driver assistance systems and autonomous driving systems, and providing alerts to the driver of a vehicle, based on anomalies detected between driver behavior and environment captured by the outward facing camera. Various aspects of the driver, which may include his direction of sight, point of focus, posture, gaze, is determined by image processing of the upper visible body of the driver, by a driver facing camera in the vehicle. Other aspects of environment around the vehicle captured by the multitude of cameras in the vehicle are used to correlate driver behavior and actions with what is happening outside to detect and warn on anomalies, prevent accidents, provide feedback to the driver, and in general provide a safer driver experience.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/275,237, filed Mar. 11, 2021, which is a U.S. National PhaseApplication of International Patent Application No. PCT/US19/50600,filed Sep. 11, 2019, which claims the benefit of and priority to U.S.Provisional Patent Application No. 62/729,994, filed Sep. 11, 2018, theentire contents of which are hereby incorporated by reference in theirentireties.

BACKGROUND Field

Certain aspects of the present disclosure generally relate tointelligent driving monitoring systems (IDMS), driver monitoringsystems, advanced driver assistance systems (ADAS), and autonomousdriving systems, and more particularly to systems and methods fordetermining and/or providing reporting to the aforementioned systemsand/or alerts to an operator of a vehicle.

Background

Vehicles, such as automobiles, trucks, tractors, motorcycles, bicycles,airplanes, drones, ships, boats, submarines, and others, are typicallyoperated and controlled by human drivers. Through training and withexperience, a human driver may learn how to drive a vehicle safely andefficiently in a range of conditions or contexts. For example, as anautomobile driver gains experience, he may become adept at driving inchallenging conditions such as rain, snow, or darkness.

Drivers may sometimes drive unsafely or inefficiently. Unsafe drivingbehavior may endanger the driver and other drivers and may risk damagingthe vehicle. Unsafe driving behaviors may also lead to fines. Forexample, highway patrol officers may issue a citation for speeding.Unsafe driving behavior may also lead to accidents, which may causephysical harm, and which may, in turn, lead to an increase in insurancerates for operating a vehicle. Inefficient driving, which may includehard accelerations, may increase the costs associated with operating avehicle.

The types of monitoring available today, however, may be based onsensors and/or processing systems that do not provide context to atraffic event. For example, an accelerometer may be used to detect asudden deceleration associated with a hard-stopping event, but theaccelerometer may not be aware of the cause of the hard-stopping event.Accordingly, certain aspects of the present disclosure are directed tosystems and methods of driver monitoring that may incorporate context aspart of detecting positive, neutral, or negative driving actions.

SUMMARY

Certain aspects of the present disclosure provide a method. The methodincludes capturing, by at least one processor of a computing device withan outward facing camera, first visual data of an outward scene outsideof a vehicle. The method further includes determining, by the at leastone processor based on the first visual data, a potentially unsafedriving condition outside of the vehicle and an amount of time in whichthe vehicle will encounter the potentially unsafe driving condition. Themethod further includes capturing, by the at least one processor with adriver facing camera, second visual data of a driver of the vehicle. Themethod further includes determining, by the at least one processor basedon the second visual data, whether a direction in which the driver islooking is toward to the potentially unsafe driving condition or awayfrom the potentially unsafe driving condition. The method furtherincludes transmitting, by the at least one processor to a remote server,a remote alert in response to determining the potentially unsafe drivingcondition and the direction in which the driver is looking such that:when the driver is determined to be looking away from the potentiallyunsafe driving condition the remote alert is transmitted in response todetermining that the amount of time in which the vehicle will encounterthe potentially unsafe driving condition is at or below a firstthreshold of time, when the driver is determined to be looking towardthe potentially unsafe driving condition the remote alert is transmittedin response to determining that the amount of time in which the vehiclewill encounter the potentially unsafe driving condition is at or below asecond threshold of time, and the first threshold of time is greaterthan the second threshold of time.

Certain aspects of the present disclosure provide a method. The methodincludes capturing, by at least one processor of a computing device withan outward facing camera, first visual data of an outward scene outsideof a vehicle. The method further includes determining, by the at leastone processor based on the first visual data, a potentially unsafedriving condition outside of the vehicle and an amount of time in whichthe vehicle will encounter the potentially unsafe driving condition. Themethod further includes capturing, by the at least one processor with adriver facing camera, second visual data of a driver of the vehicle. Themethod further includes determining, by the at least one processor basedon the second visual data, whether a direction in which the driver islooking is toward to the potentially unsafe driving condition or awayfrom the potentially unsafe driving condition. The method furtherincludes activating, by the at least one processor, an in-vehicle alertin response to determining the potentially unsafe driving condition,that the driver is looking away from the potentially unsafe drivingcondition, and that the amount of time in which the vehicle willencounter the potentially unsafe driving condition is at or below afirst threshold of time. The method further includes transmitting, bythe at least one processor to a remote server, a remote alert inresponse to a determination that the driver does not look toward thepotentially unsafe driving condition after the in-vehicle alert isactivated and that the driver does not prevent the vehicle from reachinga point where the amount of time in which the vehicle will encounter thepotentially unsafe driving condition is at or below a second thresholdof time. The first threshold of time is greater than the secondthreshold of time.

Certain aspects of the present disclosure provide a method. The methodincludes capturing, by at least one processor of a computing device withan outward facing camera, first visual data of an outward scene outsideof a vehicle. The method further includes determining, by the at leastone processor based on the first visual data, a potentially unsafedriving condition outside of the vehicle and an amount of time in whichthe vehicle will encounter the potentially unsafe driving condition. Themethod further includes capturing, by the at least one processor with adriver facing camera, second visual data of a driver of the vehicle. Themethod further includes determining, by the at least one processor basedon the second visual data, whether the driver has looked in a directionof the potentially unsafe driving condition within a predeterminedthreshold of time of the determination of unsafe driving condition. Anin-vehicle alert is suppressed when the driver has looked in thedirection of the potentially unsafe driving condition within thepredetermined threshold of time. An in-vehicle alert is activated whenthe driver has not looked in the direction of the potentially unsafedriving condition within the predetermined threshold of time.

Certain aspects of the present disclosure generally relate to providing,implementing, and using a method for determining and/or providing alertsto an operator of a vehicle. The methods may involve a camera sensorand/or inertial sensors to detect traffic events, as well analyticalmethods that may determine an action by a monitored driver that isresponsive to the detected traffic event, traffic sign, and the like.

Certain aspects of the present disclosure provide a method. The methodgenerally includes determining an indication of an inward driving scenecomplexity; adjusting at least one safety threshold based on thedetermined indication; and determining a potentially unsafe drivingmaneuver or situation based on the at least one safety threshold.

Certain aspects of the present disclosure provide a system. The systemgenerally includes a memory and a processor coupled to the memory. Theprocessor is configured to determine an indication of an inward drivingscene complexity; adjusting at least one safety threshold based on thedetermined indication; and determining a potentially unsafe drivingmaneuver or situation based on that at least one safety threshold.

Certain aspects of the present disclosure provide a non-transitorycomputer readable medium having instructions stored thereon. Uponexecution, the instructions cause the computing device to performoperations comprising determining an indication of an inward drivingscene complexity; adjusting at least one safety threshold based on thedetermined indication; and determining a potentially unsafe drivingmaneuver or situation based on that at least one safety threshold.

Certain aspects of the present disclosure provide a method. The methodgenerally includes determining an indication of an outward driving scenecomplexity; adjusting at least one safety threshold based on thedetermined indication; and determining a potentially unsafe drivingmaneuver or situation based on the at least one safety threshold.

Certain aspects of the present disclosure provide a system. The systemgenerally includes a memory and a processor coupled to the memory. Theprocessor is configured to determine an indication of an outward drivingscene complexity; adjusting at least one safety threshold based on thedetermined indication; and determining a potentially unsafe drivingmaneuver or situation based on that at least one safety threshold.

Certain aspects of the present disclosure provide a non-transitorycomputer readable medium having instructions stored thereon. Uponexecution, the instructions cause the computing device to performoperations comprising determining an indication of an outward drivingscene complexity; adjusting at least one safety threshold based on thedetermined indication; and determining a potentially unsafe drivingmaneuver or situation based on that at least one safety threshold.

Certain aspects of the present disclosure provide a system. The systemgenerally includes multiple cameras coupled to an in-vehicle computedevice comprising of memory and a processor coupled to the memory,comprising of a non-transitory computer readable medium havinginstructions stored thereon.

Certain aspects of the present disclosure provide a method. The methodgenerally includes determining keypoints on images captured by thein-vehicle camera. The keypoints may include points in the capturedimage corresponding to the Eyes, Ears, Nose, and Shoulders of thedriver. Upon detection of the keypoints, the in-vehicle compute devicemay determine gaze direction, head movements, posture of the driver, andthe like.

Certain aspects of the present disclosure provide a system. The systemgenerally includes an audio speaker device connected to the in-vehiclecompute device consisting of a processor coupled to a memory. Theprocessor is configured to activate the audio device to sound an audiblealarm to the driver upon determining anomalies in driver posture orgaze.

Certain aspects of the present disclosure provide a method. The methodgenerally includes determining deviations from straight-ahead gaze basedat least in part on images captured by the in-vehicle camera, andactivating the audio alarm when the deviations are above a predefinedthreshold.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a block diagram of an example system fordetermining, transmitting, and/or providing alerts to an operator of avehicle and/or a remote driver monitoring system in accordance withcertain aspects of the present disclosure.

FIG. 1B illustrates a front-perspective view of an example camera devicefor capturing images of an operator of a vehicle and/or an outward sceneof a vehicle in accordance with certain aspects of the presentdisclosure.

FIG. 1C illustrates a rear view of the example camera device of FIG. 1Bin accordance with certain aspects of the present disclosure.

FIG. 2 illustrates a block diagram of an example system of vehicle,driver, and/or outward scene monitoring in accordance with certainaspects of the present disclosure.

FIG. 3A illustrates a flow chart of an example method for determiningkeypoints of a driver of a vehicle and using the keypoints to determinewhether to transmit and/or activate an alert in accordance with certainaspects of the present disclosure.

FIG. 3B illustrates a flow chart of an example method for determiningobjects in an outward scene of a vehicle and using the objects todetermine whether to transmit and/or activate an alert in accordancewith certain aspects of the present disclosure.

FIG. 3C illustrates a flow chart of an example method for using visualdata captured of both the inward scene of a driver and an outward sceneof a vehicle to determine whether to transmit and/or activate an alertin accordance with certain aspects of the present disclosure.

FIGS. 4A and 4B illustrate an example of when a forward crash warning(FCW) may be transmitted and/or activated in accordance with certainaspects of the present disclosure.

FIG. 5 illustrates an example of when a warning for tired driving as aresult of yawning may be transmitted and/or activated in accordance withcertain aspects of the present disclosure.

FIG. 6 illustrates an example of when a warning for tired driving as aresult of an irregular blinking pattern may be transmitted and/oractivated in accordance with certain aspects of the present disclosure.

FIGS. 7A and 7B illustrate an example of when a warning for distracteddriving may be transmitted and/or activated in accordance with certainaspects of the present disclosure.

FIGS. 8A and 8B illustrate examples of when a warning for running a redand/or yellow light may be transmitted and/or activated in accordancewith certain aspects of the present disclosure.

FIGS. 9A-9C illustrate examples of when a warning for failing to stop ata stop sign may be transmitted and/or activated in accordance withcertain aspects of the present disclosure.

FIG. 10 illustrates example determined keypoints associated with theeyes of a driver in accordance with certain aspects of the presentdisclosure.

FIG. 11 illustrates example determined keypoints associated with an eyeand an ear of a driver in accordance with certain aspects of the presentdisclosure.

FIG. 12 illustrates example determined keypoints associated with theshoulders of a driver in accordance with certain aspects of the presentdisclosure.

FIG. 13 illustrates example determined keypoints associated with theeyes and the ears of a driver in accordance with certain aspects of thepresent disclosure.

FIG. 14 illustrates example determined keypoints associated with theeyes and the ears of a driver that is looking down in accordance withcertain aspects of the present disclosure.

FIG. 15 illustrates example determined keypoints associated with theeyes and an ear of a driver that is positioned at an angle with respectto a camera in accordance with certain aspects of the presentdisclosure.

FIG. 16 illustrates another example of determined keypoints associatedwith the eyes and an ear of a driver that is positioned at an angle withrespect to a camera in accordance with certain aspects of the presentdisclosure.

FIG. 17 illustrates example determined angles and/or distances betweenvarious keypoints associated with the eyes and an ear of a driver thatis positioned at an angle with respect to a camera in accordance withcertain aspects of the present disclosure.

FIG. 18 illustrates another example of determined angles and/ordistances between various keypoints of the driver of FIG. 17 inaccordance with certain aspects of the present disclosure.

FIG. 19 illustrates example determined angles between various keypointsof a driver including the nose, ears, and right eye of the driver inaccordance with certain aspects of the present disclosure.

FIG. 20 illustrates an example determined angle between a right ear,left ear, and left eye keypoints of a driver in accordance with certainaspects of the present disclosure.

FIG. 21 illustrates example determined angles between various keypointsof a driver including the shoulders, nose, and ears of the driver inaccordance with certain aspects of the present disclosure.

FIG. 22 illustrates an example determined angle between the ears andleft shoulder keypoints of a driver in accordance with certain aspectsof the present disclosure.

DETAILED DESCRIPTION

The detailed description set forth below, in connection with theappended drawings, is intended as a description of variousconfigurations and is not intended to represent the only configurationsin which the concepts described herein may be practiced. The detaileddescription includes specific details for the purpose of providing athorough understanding of the various concepts. However, it will beapparent to those skilled in the art that these concepts may bepracticed without these specific details. In some instances, well-knownstructures and components are shown in block diagram form in order toavoid obscuring such concepts.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the disclosure is intended to cover any aspect of thedisclosure, whether implemented independently of or combined with anyother aspect of the disclosure. For example, an apparatus may beimplemented or a method may be practiced using any number of the aspectsset forth. In addition, the scope of the disclosure is intended to coversuch an apparatus or method practiced using other structure,functionality, or structure and functionality in addition to or otherthan the various aspects of the disclosure set forth. It should beunderstood that any aspect of the disclosure disclosed may be embodiedby one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the disclosure.Although some benefits and advantages of the preferred aspects arementioned, the scope of the disclosure is not intended to be limited toparticular benefits, uses or objectives. Rather, aspects of thedisclosure are intended to be broadly applicable to differenttechnologies, system configurations, networks and protocols, some ofwhich are illustrated by way of example in the figures and in thefollowing description of the preferred aspects. The detailed descriptionand drawings are merely illustrative of the disclosure rather thanlimiting, the scope of the disclosure being defined by the appendedclaims and equivalents thereof.

Monitoring and Characterization of Driver Behavior

Driving behavior may be monitored. Driver monitoring may be done inreal-time or substantially real-time as the driver operates a vehicle,or may be done at a later time based on recorded data. Driver monitoringat a later time may be useful, for example, when investigating the causeof an accident. Driver monitoring in real-time may be useful to guardagainst unsafe driving, for example, by ensuring that a car cannotexceed a certain pre-determined speed.

The types of monitoring available today, however, may be based onsensors and/or processing systems that do not provide context to atraffic event. For example, an accelerometer may be used to detect asudden deceleration associated with a hard-stopping event, but theaccelerometer may not be aware of the cause of the hard-stopping event.Accordingly, certain aspects of the present disclosure are directed tosystems and methods of driver monitoring that may incorporate context aspart of detecting positive, neutral, or negative driving actions.

For example, aspects of the present disclosure are directed to methodsof monitoring and characterizing driver behavior, which may includemethods of determining and/or providing alerts to an operator of avehicle and/or transmitting remote alerts to a remote driver monitoringsystem. Remote alerts may be transmitted wirelessly over a wirelessnetwork to one or more servers and/or one or more other electronicdevices, such as a mobile phone, tablet, laptop, desktop, etc., suchthat information about a driver and things a driver and their vehicleencounters may be documented and reported to other individuals (e.g., afleet manager, insurance company, etc.). An accurate characterization ofdriver behavior has multiple applications. Insurance companies may useaccurately characterized driver behavior to influence premiums.Insurance companies may, for example, reward risk mitigating behaviorand dis-incentivize behavior associated with increased accident risk.Fleet owners may use accurately characterized driver behavior toincentivize their drivers. Likewise, taxi aggregators may incentivizetaxi driver behavior. Taxi or ride-sharing aggregator customers may alsouse past characterizations of driver behavior to filter and selectdrivers based on driver behavior criteria. For example, to ensuresafety, drivers of children or other vulnerable populations may bescreened based on driving behavior exhibited in the past. Parents maywish to monitor the driving patterns of their kids and may furtherutilize methods of monitoring and characterizing driver behavior toincentivize safe driving behavior.

In addition to human drivers, machine controllers are increasingly beingused to drive vehicles. Self-driving cars, for example, include amachine controller that interprets sensory inputs and issues controlsignals to the car so that the car may be driven without a human driver.As with human drivers, machine controllers may also exhibit unsafe orinefficient driving behaviors. Information relating to the drivingbehavior of a self-driving car would be of interest to engineersattempting to perfect the self-driving car's controller, to law-makersconsidering policies relating to self-driving cars, and to otherinterested parties.

Visual information may improve existing ways or enable new ways ofmonitoring and characterizing driver behavior. For example, according toaspects of the present disclosure, the visual environment around adriver may inform a characterization of driver behavior. Typically,running a red light may be considered a ‘bad’ driving behavior. In somecontexts, however, such as when a traffic guard is standing at anintersection and using hand gestures to instruct a driver to movethrough a red light, driving through a red light would be considered‘good’ driving behavior. In some contexts, a ‘bad’ driving behavior,such as tailgating, may not be the fault of the driver. For example,another driver may have pulled into the driver's lane at a potentiallyunsafe distance ahead of the driver. Visual information may also improvethe quality of a characterization that may be based on other forms ofsensor data, such as determining a safe driving speed, as describedbelow. The costs of accurately characterizing driver behavior usingcomputer vision methods in accordance with certain aspects of thepresent disclosure may be less than the costs of alternative methodsthat use human inspection of visual data. Camera based methods may havelower hardware costs compared with methods that involve RADAR or LiDAR.Still, methods that use RADAR or LiDAR are also contemplated fordetermination of cause of traffic events, either alone or in combinationwith a vision sensor, in accordance with certain aspects of the presentdisclosure.

As described herein, visual information may be further used to determinethe pose and gaze of the driver. The word “pose” is used herein to referto a sitting position, posture, and/or orientation that the driver haswhen driving a vehicle. The word “gaze” is used herein to refer to adirection where the driver is looking and/or facing.

The gaze of the driver may indicate that the driver is looking straightonto the road, or looking down at his mobile phone or looking atsomething on his right side. The pose of the driver may indicate thatthe driver is sitting in a slouched pose which may indicate drowsinessand inattentiveness. Sustained and/or periodic determinations of thepose and gaze may enable assessment and tracking and reporting ofbehavioral trends of the driver, which may inform coaching sessions,scheduling, job assignments, and the like. In some embodiments, adetermined pose and/or gaze may inform whether to alert the driverand/or safety manager about an encountered unsafe driving scenario, asdescribed in more detail below.

FIG. 1A illustrates an embodiment of the aforementioned system fordetermining and/or providing alerts to an operator of a vehicle. Thedevice 100 may include input sensors (which may include a forward-facingcamera 102, a driver facing camera 104, connections to other camerasthat are not physically mounted to the device, inertial sensors 106, carOBD-II port sensor data (which may be obtained through a Bluetoothconnection 108), and the like) and compute capability 110. The computecapability may be a CPU or an integrated System-on-a-chip (SOC), whichmay include a CPU and other specialized compute cores, such as agraphics processor (GPU), gesture recognition processor, and the like.In some embodiments, a system for determining, transmitting, and/orproviding alerts to an operator of a vehicle and/or a device of a remotedriver monitoring system may include wireless communication to cloudservices, such as with Long Term Evolution (LTE) 116 or Bluetoothcommunication 108 to other devices nearby. For example, the cloud mayprovide real-time analytics assistance. In an embodiment involving cloudservices, the cloud may facilitate aggregation and processing of datafor offline analytics. The device may also include a global positioningsystem (GPS) either as a separate module 112, or integrated within aSystem-on-a-chip 110. The device may further include memory storage 114.

A system for determining, transmitting, and/or providing alerts to anoperator of a vehicle and/or a device of a remote driver monitoringsystem, in accordance with certain aspects of the present disclosure,may assess the driver's behavior in real-time. For example, an in-carmonitoring system, such as the device 100 illustrated in FIG. 1A thatmay be mounted to a car, may perform analysis in support of a driverbehavior assessment in real-time, and may determine cause of trafficevents as they occur. In this example, the system, in comparison with asystem that does not include real-time processing, may avoid storinglarge amounts of sensor data since it may instead store a processed andreduced set of the data. Similarly, or in addition, the system may incurfewer costs associated with wirelessly transmitting data to a remoteserver. Such a system may also encounter fewer wireless coverage issues.

FIG. 1B illustrates an embodiment of a device with four cameras inaccordance with the aforementioned devices, systems, and methods ofdistributed video search with edge computing. FIG. 1B illustrates afront-perspective view. FIG. 1C illustrates a rear view. The deviceillustrated in FIG. 1B and FIG. 1C may be affixed to a vehicle and mayinclude a front-facing camera aperture 122 through which an image sensormay capture video data (e.g., frames or visual data) from the road aheadof a vehicle (e.g., an outward scene of the vehicle). The device mayalso include an inward-facing camera aperture 124 through which an imagesensor may capture video data (e.g., frames or visual data) from theinternal cab of a vehicle. The inward-facing camera may be used, forexample, to monitor the operator/driver of a vehicle. The device mayalso include a right camera aperture 126 through which an image sensormay capture video data from the right side of a vehicle operator's Pointof View (POV). The device may also include a left camera aperture 128through which an image sensor may capture video data from the left sideof a vehicle operator's POV. The right and left camera apertures 126 and128 may capture visual data relevant to the outward scene of a vehicle(e.g., through side windows of the vehicle, images appearing in siderear-view mirrors, etc.) and/or may capture visual data relevant to theinward scene of a vehicle (e.g., a part of the driver/operator, otherobjects or passengers inside the cab of a vehicle, objects or passengerswith which the driver/operator interacts, etc.).

A system for determining, transmitting, and/or providing alerts to anoperator of a vehicle and/or a device of a remote driver monitoringsystem, in accordance with certain aspects of the present disclosure,may assess the driver's behavior in several contexts and perhaps usingseveral metrics. FIG. 2 illustrates a system of driver monitoring, whichmay include a system for determining and/or providing alerts to anoperator of a vehicle, in accordance with aspects of the presentdisclosure. The system may include sensors 210, profiles 230, sensoryrecognition and monitoring modules 240, assessment modules 260, and mayproduce an overall grade 280. Contemplated driver assessment modulesinclude speed assessment 262, safe following distance 264, obeyingtraffic signs and lights 266, safe lane changes and lane position 268,hard accelerations including turns 270, responding to traffic officers,responding to road conditions 272, and responding to emergency vehicles.Each of these exemplary features is described in PCT applicationPCT/US17/13062, entitled “DRIVER BEHAVIOR MONITORING”, filed 11 Jan.2017, which is incorporated herein by reference in its entirety. Thepresent disclosure is not so limiting, however. Many other features ofdriving behavior may be monitored, assessed, and characterized inaccordance with the present disclosure.

Activating In-Vehicle Alerts and/or Transmitting Remote Alerts

FIG. 3A illustrates a flow chart of an example method 300 fordetermining keypoints of a driver of a vehicle and using the keypointsto determine whether to transmit and/or activate an alert in accordancewith certain aspects of the present disclosure. In other words, themethod 300 relates to the inward scene of a vehicle, including thedriver, where the driver is looking, what the drive is doing, etc.Frames 302 represent visual or image data captured by an inward facingcamera (e.g., through the inward-facing camera aperture 124 of thedevice of FIG. 1B). At 304, a blob for further analysis is selected formthe frames 302. For example, the image frames captured by a camera mayinclude a wider view than just a driver, but only the driver is ofinterest for analyzing the driver's pose and/or gaze. Accordingly, thesystem may use a blob to reduce the processing for analyzing a driver.

At 306, the image blob and information about the image blob (e.g., howthe original image was reshaped, resized, etc.) may be analyzed togenerate information about the driver. For example, the system maygenerate a bounding box around a driver or portion of a driver in theblob. The system may also generate coordinates of the bounding boxwithin the blob or the larger image before the blob was created. Ifthere is more than one person present inside the vehicle, more than onebounding box (one for each person) may be generated. Keypoint masks mayalso be generated about drivers. The keypoint masks are fit to theperson identified in the blob, and may be used to determine the relativecoordinates of specific keypoints with respect to the person boundingbox coordinates. In other words, the information generated about thedriver may include keypoint masks that are used to determine driverkeypoints at 308. Various types of image recognition systems may be usedto perform the steps of the method 300. For example, a deep neuralnetwork (DNN) may be used to determine whether there is a person in theblob, the person bounding box (and associated coordinates), the keypointmasks (and any associated coordinates), etc.

At 308, the keypoint masks and the driver bounding box (part of theinformation generated about the driver at 306) is used to determineindividual keypoints of the driver. As described herein, keypoints maybe used to determine pose and/or gaze of a driver. At 310, the variouskeypoints are used to determine other features/contents in the image.For example, the identified keypoints may indicate where a seatbelt is,where a part of the driver is (e.g., eyes, shoulders, nose, mouth, head,arms, hands, chest, etc.), etc. The keypoints themselves and/or thefeatures/contents identified in the image/visual data may be used todetermine pose, gaze, and or other aspects (e.g., is seatbelt on, isdriver wearing sunglasses, is driver holding something, etc.). Boundingboxes with associated coordinates for each identified feature/contentmay also be generated by the system, such that those features/content ofthe image as identified may be monitored by the system.

In various embodiments, a model that recognizes and tracks features of adriver may also recognize objects within the vehicle, such as asmartphone, drink cup, food, phone charger, or other object. If anobject is determined in the inward scene, the location information ofthat object may be used to determine whether the driver is distracted ornot. For example, if a driver holds up their smartphone so that it ispart of their field of view out the windshield, the system may see thedriver as looking forward properly. However, the presence of thesmartphone elevated into the field of view of the windshield mayindicate distracted driving. Accordingly, if the system determines thatthe driver is looking ahead but the smartphone is elevated in field ofview for a particular threshold of time and frames over that time, thesystem may determine that a driver is distracted or otherwise notlooking at a potentially unsafe condition outside of the vehicle andtrigger alerts accordingly. A smartphone may be determined, for example,by determining a wrist keypoint of the driver, cropping around the wristand classifying the region around the wrist with a phone detection thatlooks for the shape and/or edges of a smartphone. Once the location ofthe phone is known, it may be used in conjunction with pose and gazeinformation to determine if the driver is looking at the phone.

Over time, the features/contents identified at 310 may be monitored, anddifferent frames classified to determine what a driver is doing overtime. In other words, at 312, the features/contents are accumulated overtime and their characteristics are determined so that the system mayunderstand what the driver is doing, looking at, feeling, etc. Forexample, a seatbelt bounding box may be classified as absent (notfastened on driver) or present (fastened on driver). If a seatbelt notpresent is accumulated over a predetermined threshold number of frameswhile the vehicle is being operated, for example, an alert may beactivated in-vehicle and/or may be transmitted to a remote server. Inother examples, a yawn may be detected by accumulating classificationsof a mouth of open, closed, or not sure. If a mouth is classified asopen over a certain number of image frames that coincides with a typicalamount of time for a yawn, the driver may be considered to have yawned.Eyes may be monitored to detect blinks, long blinks or other eyeclosures that may indicate a driver falling asleep, glasses on with eyesopen, glasses on with eyes closed (e.g., for detecting blinks or othereye closures), glasses on with eyes not visible, or not sure. If, forexample, an eye closure is detected over a predetermined amount of time(e.g., corresponding to a particular number of frames), the system maydetermine that the driver is falling asleep.

The system may also detect pose and gaze to determine the posture of adriver and/or where the driver is looking. The pose and gaze informationmay also be accumulated to determine if a driver is distracted bysomething for longer than a threshold amount of time, to determine if adriver is looking at or has recently looked at something (e.g., isdriver looking at a potentially unsafe driving condition such asapproaching a red light without slowing down, has driver recently lookedin mirror and/or shoulder checked adjacent lane before changing lanes,etc.). The predetermined thresholds of time for accumulating featuresmay differ before any action is taken for various features. For example,if a blink lasts more than two or three seconds an alert may beactivated in-vehicle and/or transmitted remotely. A yawn may bedetermined to have occurred where the mouth is open for, e.g., threeseconds. In another example, an alert relating to a seatbelt may not betriggered until the system has determined that the driver has not beenwearing a seatbelt for one minute. Accordingly, at 316, in-vehiclealerts may be activated and/or remote alerts may be transmitted based onaccumulated features as described herein. In various embodiments, thepredetermined thresholds for whether to activate an in-vehicle alert maybe different than the accumulation thresholds for transmitting a remotealert. In some examples, the threshold for whether to activate anin-vehicle alert may be shorter, and the system may determine if thedriver responds to the alert. If the driver does not respond to thein-vehicle alert, the system may transmit the remote alert after asecond, longer threshold of time has accumulated with respect to adetected feature. As described herein, any of the information collectedabout the inward scene (e.g., of the driver) of a vehicle may be used inconjunction with information about the outward scene of the vehicle todetermine when and if to activate and/or transmit alerts.

At 314, the system may use various accumulated features (e.g.,shoulders, head, arms, eyes, etc.) to determine the pose and/or gaze ofthe driver. In other words, the various keypoints, feature boundingboxes, etc. may be used to detect where the driver is looking and/or theposture of the driver over time. For example, the system may calibrate anormal pose and/or gaze of the driver as further described herein. Thatinformation may be used to feed back into 310 to determine a normal poseand/or gaze of the driver based on the various keypoints/bounding boxesbeing monitored. Then the system can accumulate various featuredetections at 312 after the pose and/or gaze calibration is complete todetermine deviations from a normal pose and/or gaze over time. In otherwords, the system may compute normalized distances, angles, etc. of aparticular driver so that the system can determine when thosemeasurements change to determine looking down, looking right, lookingleft, etc. Gaze and pose detection is further demonstrated describedherein, including with respect to FIGS. 10-22 .

In various embodiments, thresholds for a number or percentage ofaccumulated features detected over a particular time threshold may alsobe utilized. For example, if a driver has their eyes closed, the systemmay not be able to detect that the driver's eyes are closed for everysingle frame captured over the course of, e.g., three seconds. However,if the system detects eye closure in, e.g., 70% of frames captured overthree seconds, the system may assume that the driver's eyes were closedfor all three seconds and activate or transmit an alert. Detections maynot be perfectly accurate where, for example, a frame is saturated dueto sunlight, a frame is too dark, the driver has changed pose sosignificantly that the normal features/keypoints/bounding boxes may notbe useful for accumulating feature detections, etc. Other thresholds maybe used. For example, an alert may be transmitted or activated if aseatbelt is detected on the driver less in less than 30% of frames overthe course of a minute. An alert may be transmitted or activated if agaze of a driver is determined such that the driver is looking down in95% or more of frames captured over the course of three seconds.

Other rules, thresholds, and logic may also be used at 316 to determinewhether and/or when to activate and/or transmit an alarm. For example,aspects of the vehicle may be taken into account. For example, certainalarms may not be triggered if the vehicle is going less than apredetermined threshold of speed (e.g., five miles per hour (mph)), evenif an accumulated feature would otherwise indicate triggering an alarm.In another example, an alarm may be suppressed if, for example, afeature that relies on a certain orientation of the driver is notoccurring. For example, if a driver is looking left to check an adjacentlane, the driver's eyes may not be visible to determine blinks.Accordingly, if the driver is not looking straight, the system mayautomatically not accumulate any eyes closed determinations for purposesof triggering alarms.

FIG. 3B illustrates a flow chart of an example method 330 fordetermining objects in an outward scene of a vehicle and using theobjects to determine whether to transmit and/or activate an alert inaccordance with certain aspects of the present disclosure. At 334 a blobis created using frames 332. In various embodiments, blob creation maynot be performed where everything a camera captures outward of thevehicle is potentially relevant. In other embodiments, the camera maycapture some things that are not relevant to an outward scene, such aspart of the vehicle in which the camera is mounted.

At 336, captured image frames are analyzed and information about objectsin the image is generated. At 338, the coordinates/locations of objectsin the images may be determined. The coordinates/locations of objects inthe images may be determined, for example, by applying masks to theimage to find other vehicles, traffic control devices, lanes, curbs,etc. Bounding boxes may be generated for those objects, and furtherprocessing of the image may be performed at 340 to determine theidentity and location of objects in the images. For example, the typesof signs detected may be determined, the location and identity ofvehicles may be determined, etc. At 342, the detected objects areaccumulated over time. For example, other vehicles may be monitored overtime to determine, e.g., how close the other vehicle is to the vehiclewith the camera. Information accumulated about objects detected in theoutward scene may be used to determine whether to transmit remote alertsand/or activate in-vehicle alerts at 344 as described herein. Forexample, if the vehicle with the camera is rapidly approaching a stoppedvehicle in the road, the system may determine that an in-vehicle alertmay be activated. The method 330 may also be used in conjunction withthe method 300 with a set of rules and logic such that alerts use bothinward and outward scene information. For example, an in-vehicle alertmay be activated sooner if the driver's gaze indicates that the driveris not looking toward the potentially unsafe driving condition (e.g.,the stopped vehicle in the road), or has not looked toward thepotentially unsafe driving condition within a threshold of time.

FIG. 3C illustrates a flow chart of an example method 350 for usingvisual data captured of both the inward scene of a driver and an outwardscene of a vehicle to determine whether to transmit and/or activate analert in accordance with certain aspects of the present disclosure. At352, first visual data of an outward scene outside of a vehicle iscaptured with an outward facing camera. At 354, a potentially unsafedriving condition outside of the vehicle and an amount of time in whichthe vehicle will encounter the potentially unsafe driving condition isdetermined based on the first visual data. At 356, second visual data ofa driver of the vehicle is captured with a driver facing camera. At 358,the system determines, based on the second visual data, whether adirection in which the driver is looking is toward to the potentiallyunsafe driving condition or away from the potentially unsafe drivingcondition. At 360, an in-vehicle alert is activated based on thedirection the driver is looking and/or the amount of time in which thevehicle will encounter the potentially unsafe driving condition. At 362,a remote alert is transmitted, to a remote server, based on thedirection the driver is looking, the amount of time in which the vehiclewill encounter the potentially unsafe driving condition, and/or whetherthe driver responds to the in-vehicle alert. As described hereinthroughout, various combinations of in-vehicle and remote alerts may beactivated/transmitted in different situations, including the severity ofthe incident, whether the driver responded timely to an in-vehiclealert, the type of potentially unsafe driving condition that occurred,etc. In various embodiments, an in-vehicle alert may not be activatedand a remote alert may still be transmitted. In various embodiments, anin-vehicle alert may be activated while a remote alert is nottransmitted.

For example, a remote alert and/or the in-vehicle alert may be triggeredwhen the driver is determined to be looking away from the potentiallyunsafe driving condition and in response to determining that the amountof time in which the vehicle will encounter the potentially unsafedriving condition is at or below a first threshold of time. The remotealert and/or the in-vehicle alert may also be triggered when the driveris determined to be looking toward the potentially unsafe drivingcondition. The remote alert is transmitted in response to determiningthat the amount of time in which the vehicle will encounter thepotentially unsafe driving condition is at or below a second thresholdof time. The first threshold of time in this example may be greater thanthe second threshold of time, such that an alert is triggered morequickly if the driver is not looking toward the potentially unsafecondition.

In another example, the in-vehicle alert may be activated before theremote alert is transmitted (e.g., the predetermined thresholds of timeassociated with the in-vehicle alert and the remote alert aredifferent). In this way, the driver may have a chance to respond to thealert and remedy the potentially unsafe driving condition before theremote alert is transmitted. In other words, the remote alert may besent in response to a determination that the driver does not look towardthe potentially unsafe driving condition after the in-vehicle alert isactivated and/or that the driver does not prevent the vehicle fromreaching a point where the amount of time in which the vehicle willencounter the potentially unsafe driving condition is at or below apredetermined threshold of time. Accordingly, four different amount oftime thresholds may be used: 1) in-vehicle alert for when driver islooking at potentially unsafe condition, 2) in-vehicle alert for whendriver is not looking at the potentially unsafe condition, 3) remotealert transmission for when driver is looking at potentially unsafecondition, and 4) remote alert transmission for when the driver is notlooking at the potentially unsafe condition.

The remote alert transmission may include various types of information,data, the images or video associated with the alert (from inside thevehicle and/or the outward scene), etc. The information in the remotealert may also include information about the determined pose and gaze ofthe driver at and before the remote alert transmission is made,including any accumulated pose/gaze information, rules triggered,exceptions, etc. The amount of time in which a vehicle with a camerawill encounter the potentially unsafe driving condition is determinedbased on at least one of a speed of the vehicle, a distance from thevehicle to an object associated with the potentially unsafe drivingcondition, and/or a speed of the object associated with the potentiallyunsafe driving condition. The object associated with a potentiallyunsafe driving condition may include any of a traffic light, a stopsign, an intersection, a railroad crossing, a lane or road boundary, asecond vehicle, lane or road boundary, or any other object, obstruction,etc.

In various embodiments, when a remote alert is transmitted, a remotedevice or party may be able to request and/or otherwise activate a livevideo feed from one or more of the cameras in the vehicle. For example,if a driver is falling asleep as determined by the systems and methodsdescribed herein, the monitoring device in the vehicle may send a remotealert to remote server. A fleet manager, for example, may receive theremote alert, watch recorded video associated with the alert. The remotealert may include an option, presented to the fleet manager through agraphical user interface (GUI), to request a live video feed from thein-vehicle monitoring device. Accordingly, a request to stream livevideo captured by at least one of an outward facing camera or a driverfacing camera is sent to the in-vehicle device, and the in-vehicledevice may begin transmitting the live video in response to the requestback to a device of the fleet manager. Each of the inward and outwardcamera videos may be streamed, or the fleet manager may select, throughthe GUI, which camera feed to stream.

FIGS. 4-9 demonstrate various examples of outward and inward scenes of avehicle that may be monitored. The outward scenes may includepotentially unsafe driving conditions as described herein, and theinward scenes may include various driver behavior that may be determinedusing pose and/or gaze determinations as described herein. In addition,the determinations made about the inward and outward scenes of a vehiclemay be used in combination to determine when and whether to activatein-vehicle alarms and/or transmit remote alarms as described herein.

FIGS. 4A and 4B illustrate an example of when a forward crash warning(FCW) may be transmitted and/or activated in accordance with certainaspects of the present disclosure. In FIG. 4A, a vehicle 402 is in theoutward scene in front of the vehicle with the camera. The distancebetween the two vehicles and the speed at which the vehicles aretraveling may be used to determine the amount of time in which thevehicle with the camera will encounter the vehicle 402, a potentiallyunsafe driving condition. If the amount of time dips below a particularthreshold, for example as the vehicle 402 gets closer in FIG. 4B, analert may be triggered. For example, the threshold may be two seconds.

FIG. 5 illustrates an example of when a warning for tired driving as aresult of yawning may be transmitted and/or activated in accordance withcertain aspects of the present disclosure. FIG. 5 shows a driver 502with a mouth 504 that is open, which may indicate yawning as describedherein. FIG. 5 also shows a bounding box 506 for the driver, and ablurred area 508. The bounding box 506 may represent a blob that isdetermined to have the driver in it, so that the area 508 outside thebounding box 506 need not be analyzed for a driver's pose, gaze,keypoints, etc. The bounding box 506 may also be used to determinevarious keypoints, such as the top of the driver 502's head, the driver502's shoulders, etc.

FIG. 6 illustrates an example of when a warning for tired driving as aresult of an irregular blinking pattern may be transmitted and/oractivated in accordance with certain aspects of the present disclosure.FIG. 6 shows a driver 604 in a bounding box 602, and eyes 606 that areclosed. As described herein, eyes closed determinations may beaccumulated to determine an irregular blinking pattern or eyes closedfor a long period of time or there is abnormally fast blinking thatcould indicate sleeping and/or tiredness in the driver 604.

FIGS. 7A and 7B illustrate an example of when a warning for distracteddriving may be transmitted and/or activated in accordance with certainaspects of the present disclosure. In FIG. 7A, a driver 702 looks to hisleft out the window at a first time 704. The looking away may be markedby a symbol 706 in the timeline. Then, at a second time 708 after thefirst time 704, the driver 702 is still looking to his left out thewindow (compare how the second time 708 has elapsed from the symbol 706position). As described herein, the looking left of the driver 702 maybe accumulated to determine if the driver is distracted. In otherembodiments, if the driver 702 looked back forward at the second time708, no alert may be triggered. In various embodiments, determinationsof looking left or right may be used to suppress an alert instead oftrigger an alert. For example, if a driver turns on their blinker and/orbegins to change lanes, an alert may be triggered if the driver has notlooked toward the lane in which they are changing into within apredetermined threshold of time within activating the blinker and/orbeginning to move into the other lane. In contrast, if the driver haslooked into the other lane and/or a mirror associated with the otherlane within a predetermined amount of time, the user will be determinedto not be distracted and no alarm may be triggered.

FIGS. 8A and 8B illustrate examples of when a warning for running a redand/or yellow light may be transmitted and/or activated in accordancewith certain aspects of the present disclosure. FIG. 8A demonstrates avehicle in an intersection 804 when a traffic light 802 turns or isyellow. In such an embodiment, an alert may be triggered after thepotentially unsafe condition is determined. For example, a warning notto enter intersections when a traffic light is yellow may be activatedin-vehicle for the driver. Such a situation may be considered a minorviolation because no red light was run. In various embodiments, theclassification of an event as minor or major may cause different alerts.For example, a minor event (running yellow light) may not trigger aremote alert. In other examples, a remote alert may be triggered ifthere are a threshold number of minor events over a particular amount oftime that indicates a driver taking too many risks. In some embodiments,the system may not be able to activate an in-vehicle alert an adequateamount of time before the potentially unsafe driving condition occursfor the driver to avoid (e.g., running a yellow light). In suchembodiments, alerts may merely be sent to warn the driver for the futureor remotely monitor driving. In FIG. 8B, an example of a major trafficviolation is shown, where the traffic light 806 is red before thevehicle enters an intersection 808. Because, the traffic light 806 isalready red, an in-vehicle alert may be activated in time to warn thedriver (e.g., if the driver is not looking toward the traffic light) sothat the vehicle might be stopped before entering the intersection 808.

FIGS. 9A-9C illustrate examples of when a warning for failing to stop ata stop sign may be transmitted and/or activated in accordance withcertain aspects of the present disclosure. Stop sign violations may alsobe categorized as major or minor, depending on the speed, context (e.g.,are other vehicles/pedestrians present?), etc. For example, if a vehicleslows down under five mph that may be characterized as a minor rollingstop violation. If a vehicle completely stops or gets under some otherspeed threshold (e.g., 2 mph), the driver may be determined to havecompletely stopped. The system may determine that a potentially unsafedriving condition about which to trigger an alert is a stop sign wherethe driver is looking away and/or a stop sign where the systemdetermines that the vehicle is not slowing down at a rate that will leadto at least a rolling stop or a complete stop. FIG. 9A demonstrates avehicle that may be one second from a rolling stop violation because thevehicle is close to the stop sign 902, is travelling 15 mph, and isslowing down. FIG. 9B demonstrates a vehicle that may be one second froma full, major no stop violation because the vehicle is close to the stopsign 904, is travelling 15 mph, and is either speeding up or maintaininga constant speed. This information may be used along with the inwardscene of a vehicle and the pose/gaze of a driver to determine when andwhether to trigger alerts as described herein. FIG. 9C shows a stop sign906 that is determined not to be applicable to the vehicle with thecamera. Any of the bounding box, shape of the edges, location within theimage, etc. may be used to determine that the stop sign 906 does notapply to the vehicle, therefore any alerts associated with the stop sign906 may be suppressed or otherwise not triggered.

Gaze and Pose Detection

In an embodiment of certain aspects of the present disclosure, machinelearning (ML) algorithms that may include neural networks, such asConvolutional Neural Networks, may be used to detect keypoints relatedto a driver of a vehicle. Detected keypoints may correspond to locationsin visual data corresponding to one or more of the following: a left earof the driver, a right ear of the driver, a left eye of the driver, aright eye of the driver, a nose of the driver, a left shoulder of thedriver, a right shoulder of the driver. Other keypoints are alsocontemplated.

Convolutional Neural Networks (CNNs) are a class of Neural Network (NN)that may be applied to visual imagery. Because convolutional kernelsusually applied to different locations of an input image, a givenconvolutional kernel may learn to detect one or more salient visualfeatures at substantially any location in the image. By convolving akernel with input data in a degree of translational invariance inkeypoint detection may be achieved. Alternatively or in addition, otherNeural Network architectures may be employed. In one example, aFully-Connected or Locally-Connected Neural Network may be employed. Insome embodiments, a Neural Network may comprise one or more layershaving a convolutional kernel and one or more layers having afully-connected layer. Unlike a convolutional layer of a neural network,a Fully-Connected or Locally-Connected neural network layer may beexpected to process different portions of the input image with differentkernels in different locations. Likewise, a Fully-Connected orLocally-Connected neural network layer may be expected to processdifferent feature map inputs from upstream layers in a manner thatvaries across the feature map.

In some embodiments, such as in an embodiment directed to anafter-market product, there may be a need achieve a high degree oftranslational invariance, as this may then support a wider range ofmounting positions, camera lens properties, and the like. Accordingly,it may be desirable to detect keypoints of a driver wherever the drivermay appear in visual data. Because there may be a high expected varianceacross installations of such an after-market product, convolutionalkernels may be effectively employed to achieve a desired translationalinvariance.

A set of images with labeled keypoints may be referred to as trainingdata. The training data may be provided as input to an ML algorithm,such as an ML algorithm configured to train a neural network to processvisual data. In one example, the labeled keypoints may be represented bya one-hot encoding in which the target pixel location is representedwith a number corresponding to the category of the keypoint and allother pixel locations are represented as zeros. In another embodiment,the pixel location of the labeled keypoints may be represented withoutregard to the category of the keypoint and the category of the keypointmay be determined separately. After processing image data, lossgradients may be applied to weights in the neural network that wouldhave reduced the error on the processed data. Over repeated iterations,these techniques may train the neural network to detect features aroundthe labelled keypoints that are important for detecting these keypoints.

Once the system learns from a set of training data (a set of images withlabelled keypoints), the system may be tested to ensure that it is ableto detect the keypoints from a different set of images. This differentset of images may also have labeled keypoints available and the set ofimages and labeled keypoints may be referred to as test data. The errorsfrom test data may be used to determine when training should stop. Thetesting data may be considered distinct from the training data, however,because the errors calculated on the test data may not be applied toupdate neural network weights directly. By maintaining this distinction,the performance of the neural network outputs may be more likely togeneralize to images that are not present in the training or testingdata, because the test data may be considered a proxy for data that theneural network may encounter after it is deployed. These two steps oftraining and testing may be repeated with random subsets of the trainingdata until the accuracy of the neural network on the test data reaches adesired level of accuracy.

Certain aspects of the present disclosure provide a method to normalizethe distance between detected keypoints. In one embodiment of certainaspects of the present disclosure, the distance between the left eye ofthe driver and the right eye of the driver, may be normalized by thedistance between the left shoulder of the driver and the right shoulder.As illustrated in detail below, the shoulder distance of the driver maybe an average or median distance between a first keypoint correspondingto the left shoulder of the driver and a second keypoint correspondingto the right shoulder of the driver. As explained below, the medianvalue of this distance may correspond to the distance between thedriver's shoulders when the driver is seated in a normal drivingposition (a typical driving pose).

In this first example, the determined keypoints that may be used tocalculate the median shoulder distance may be continuously orperiodically calculated from captured images of the driver. The medianvalue of the shoulder distance may be calculated from all of thecollected shoulder distance values over a pre-configured time interval.In one embodiment, the pre-configured time interval may be 2 minutes. Bycalculating the median of the shoulder distance determinations, thesystem may converge on a shoulder distance that corresponds to thedriver in a typical driving posture.

According to certain aspects of the present disclosure. the medianshoulder distance thus calculated may then be applied to normalize oneor more determined distances between other pairs of keypoints. Forexample, if the driver leans forward thus coming closer to the camera,the distance between the left eye and the right eye (eye distance),which is the distance between the keypoint of the left eye and thekeypoint of the right eye will increase in the captured image becausethe driver's head will occupy more of the image frame. The shoulderdistance will likewise increase in this captured image. Since the systemhas calculated the median shoulder distance that corresponds to atypical pose, it may now use that value to determine a scaling factorbetween the median shoulder distance and the shoulder distancedetermined in the current frame. This scaling factor, in turn, may beused to scale the eye distance observed in the same frame. Methods fordetecting gaze changes that are based on these scaled keypoint distancesas disclosed herein may then be more robust to temporary posturalchanges than are methods that do not include such a normalizing step.Likewise, normalizing a keypoint distance by another determined mediankeypoint distance, as disclosed herein, may improve robustness tovariations in the relative positions of the camera and the driver.

Accordingly, certain aspects of the present disclosure are directed toenabling the use of visual data of the driver facing camera in thevehicle to accurately detect the gaze of the driver as well as changesof the driver's gaze. While there are existing systems for determiningthe gaze of a driver, these systems may only work acceptably well for acamera located in a known position and for a driver who is seated withina relatively narrow range of distances from the camera. That is, withoutthe benefit of certain aspects of the present disclosure, a determinedgaze direction of two people, each situated in a different automobileand who are looking in the same direction outside of their respectiveautomobile, may differ if those two people are of different heights ordrivers may adjust their seats differently. In contrast, a system thatis enabled with the present teachings may learn a median keypointdistance, such as a shoulder distance, of each driver. The system maythen use the median keypoint distance normalize other keypointdistances, and therefore overcome this shortcoming of currentlyavailable gaze detection systems.

A median shoulder keypoint distance of a driver may be saved in anin-vehicle monitoring device or on a storage device in the cloud. Thisdata may be retrieved by the monitoring device the next time the samedriver is driving this vehicle. The retrieved shoulder keypoint distancemay be used to normalize other keypoint distances immediately. In thisway, the system may avoid a calibration period, such as thepreconfigured amount of time described above, during which it isexpected to find the median shoulder distance. In some embodiments, themedian shoulder keypoint distance may be updated periodically, such asdaily or weekly.

In one embodiment, the driver facing camera continuously captures imagesof the driver and transmits a subset of the images for processing on theonboard compute device. The visual data from the driver facing camerasensor is the image of the driver that is continuously received at thecamera. This may be a preconfigured number of times, say 5 frames persec (fps). This image data may be processed at the connected computedevice next to the camera in the vehicle. The compute device may inaddition send this data to another compute server in the cloud, whichmay have a more powerful graphics processor (GPU), digital signalprocessor (DSP), or other hardware accelerator.

Pose and gaze detection may be based on a sequence of object detectionsfrom more than one video frame (image). In some embodiments, the objectdetections across multiple frames may be used to infer the changes ofpose and gaze of the driver and gain confidence in the detection by thecompute device in the vehicle.

FIG. 10 illustrates an embodiment of the aforementioned devices, systemsand methods for determining alerts based on visual data fromdriver-facing and outward-facing visual data. In this illustration, thedriver's head may be visible in the image data captured by thedriver-facing camera. The system may detect a first keypoint 1001corresponding to the right eye of the driver and a second keypoint 1002corresponding to the left eye of the driver. From the images captured bythe driver-facing camera, the compute device may determine an ‘eyedistance’ 1003 of the driver. As described above with respect to amedian shoulder distance, a median ‘eye distance’ may be determined bysampling multiple images in a preconfigured time interval. In oneembodiment, if the camera is capturing at 5 frames per second (FPS), andthe pre-configured time is 2 minutes, then, the camera would capture5*60(sec)*2 (min)=600 images. In some embodiments, the camera maycapture frames at a higher frame rate, but may send a subset of thecaptured frames to the compute device for processing. For example, ifevery sixth frame is processed from a camera that captures 30 FPS, theeffective processing rate may be 5 FPS. From each of these processedimages, a set of keypoints, with a first keypoint 1001 corresponding tothe right eye and a second keypoint 1002 corresponding to the left eye,may be detected. The eye distance 1003 may be calculated for each framein which both the right eye keypoint 1001 and the left eye keypoint 1002are detected.

While the above examples describe using keypoints associated with theshoulders and/or the eyes, other embodiments are also contemplated. Apair of keypoints that may be used to determine a median keypointdistance may be associated with a variety of points on the face of thedriver or on the body of the driver.

In another embodiment and referring to FIG. 11 , a system in accordancewith certain aspects of the present disclosure may be configured todetermine a median ‘ear to eye’ angle. Depending on which ear of thedriver is detectable in more image frames, which would depend on theside of the vehicle on which the driver drives and the location of thecamera, the left or right ear may be used. In one example, the ‘ear toeye’ angle may be the angle 1105 that is formed by a first line 1103(that connects the left ear keypoint 1106 and the right ear keypoint1101) and a second line 1104 (the connects the right eye keypoint 1102and the right ear keypoint 1101).

FIG. 12 illustrates an embodiment of the aforementioned devices, systemsand methods for determining alerts based on visual data fromdriver-facing and outward-facing visual data. In this illustration, thedriver's head may be visible in the image data captured by thedriver-facing camera. The system may detect a first keypoint 1202corresponding to the right shoulder of the driver and a second keypoint1201 corresponding to the left shoulder of the driver. From the visualdata captured by the driver-facing camera, the compute device maydetermine a shoulder distance 1203 of the driver in each frame.Furthermore, as described above, by sampling multiple images in apreconfigured time interval, a median shoulder distance may bedetermined.

Additional keypoint distances are contemplated and may be useful forembodiments of the aforementioned devices, systems and methods fordetermining alerts based on visual data from driver-facing andoutward-facing visual data. In one example and referring to FIG. 13 , an‘ear to ear’ keypoint distance may be calculated between a left earkeypoint 1304 and a right ear keypoint 1303. Alternatively, or inaddition, an ‘eye to eye’ keypoint distance may be calculated between aright eye keypoint 1301 and a left eye keypoint 1302.

Furthermore, a ‘nose to left ear’ keypoint distance may be determinedbased on a nose keypoint 1306 and a left ear keypoint 1304. In anexample, this would be the length of the line drawn from the keypoint1306 to the keypoint 1304. Likewise, a ‘nose to right ear’ keypointdistance may be determined based on a nose keypoint 1306 and a right earkeypoint 1303.

In another embodiment and referring to FIGS. 17 and 18 , a system inaccordance with certain aspects of the present disclosure may beconfigured to determine a first keypoint 1805 corresponding to the noseof the driver and a second keypoint 1807 corresponding to the left earof the driver. In one example, the ‘nose to left ear’ angle may be theangle 1802 that is formed by a first line 1803 (that connects the nosekeypoint 1805 and the driver facing camera 1801) and a second line 1804(that connects the left ear keypoint 1807 and the driver facing camera1801). From the visual data captured by the driver-facing camera, thecompute device will be able to determine the position of the camera1801. Similarly, in FIG. 17 , a first keypoint 1701 may be defined withrespect to the right eye and a second keypoint 1702 may be defined withrespect to the left eye. A third keypoint 1703 may be defined withrespect to the driver's left ear. The driver may be captured with acamera 1706. A ‘left ear to left eye’ angle may be an angle 1704 that isformed by a first line between the camera 1706 to the third keypoint1703 and a second line between the camera 1706 to the second keypoint1702. A ‘left eye to right eye’ angle may be an angle 1705 that isformed by a third line between the camera 1706 and the first keypoint1701 and the second line. A ‘left ear to right eye’ angle may be anangle formed by the first line and the third line (e.g., the angle 1704plus the angle 1705).

In another example and referring to FIG. 19 , a system in accordancewith certain aspects of the present disclosure may be configured todetermine the ‘nose to right ear’ angle 1902 that is formed by a firstline 1903 (that connects the right ear keypoint 1905 and the driverfacing camera 1901) and a second line 1904 (that connects the nosekeypoint 1906 and the driver facing camera 1901).

In another example, a system in accordance with certain aspects of thepresent disclosure may be configured to determine a keypoint angle 1909that is subtended at the right ear keypoint and formed between the 2lines, the first line 1907 (that connects the right ear keypoint 1905and the right eye keypoint 1910) and the second line 1908 (that connectsthe right ear keypoint 1905 and the left ear keypoint 1906).

In another example and referring to FIG. 20 , a keypoint angle may beangle 2001 that is subtended at the left ear keypoint and formed by afirst line 2003 (that connects the left ear keypoint 2004 and the lefteye keypoint 2005) and a second line 2002 (that connects the left earkeypoint 2004 and the right ear keypoint 2006).

In another example and referring to FIG. 21 , a keypoint angle may beangle 2105 that is subtended at the right shoulder keypoint 2102 andformed by a first line 2106 (that connects the right shoulder keypoint2102 and the right ear keypoint 2107) and a second line 2104 (thatconnects the right shoulder keypoint 2102 and the nose keypoint 2103).

In another example referring to FIG. 21 , a keypoint angle may be angle2108 that is subtended at the left shoulder keypoint 2101 and formed bya first line 2109 (that connects the left shoulder keypoint 2101 and theleft ear keypoint 2110) and a second line 2111 (that connects the leftshoulder keypoint 2101 and the nose keypoint 2103).

In another example and referring to FIG. 22 , a keypoint angle may beangle 2205 that is subtended at the left shoulder keypoint 2201 andformed by a first line 2203 (that connects the left shoulder keypoint2201 and the right ear keypoint 2207) and a second line 2204 (thatconnects the left shoulder keypoint 2201 and the left ear keypoint2206).

In one embodiment, the above-mentioned angles and distances betweenkeypoints are arranged in a sorted list, for the compute device to findthe median of each of these calculated distances and angles. As anexample, the compute device will determine the median for shoulderdistance, a median for the eye distance, etc. In certain embodiments,there may be more keypoints that are captured from the images. Theembodiment describe above is an example of a few keypoints to help inthe explanation. In one embodiment, these median values are calculatedcontinuously for every 2-minute interval. The compute device havingfound the median of these various values, records these median values as“effective distance or effective angle”, for each of the 2-minutesamples. This data may be also sent to the remote cloud based server andsaved against this driver profile in a database.

In the following preconfigured time interval, which in one embodiment is2 minutes, the in-vehicle compute device on receiving the next 600images at 5 FPS from the driver facing camera, repeats the same processas above and finds a new “effective distance or effective angle” forthis next 2-minute interval.

In one embodiment, the various angles and distances between keypointsare captured once the vehicle attains a preconfigured speed, of 15 milesper hour.

In one embodiment, once the compute device has calculated the “effectivedistance or effective angle” of all the various distances and anglesbetween the keypoints in a preconfigured sample time of 2 minutes, itstarts image processing of the next 600 samples received from the camerafor the next 2 minutes of sample time. For each sample received in thisfollowing 2-minute interval, all the distances and angles are comparedto their respective “effective distances or effective angle values”,calculated in the previous 2 minute interval.

In one embodiment, the various distances and angles between keypoints,when compared to the “effective distance or effective angle” valuescalculated from the previous sample will enable the compute device todetect the pose and gaze of the driver and the driver's movementrelative to the last calculated “effective distances or effective anglevalues”. For example, if the Nose to Shoulder distance is less than acertain factor compared to the “effective distance” of the Nose toShoulder from the previous time period, it indicates that the driver islooking down.

FIG. 13 shows the keypoints to calculate the vertical distance betweenthe eyes and ears. In this illustration, the driver's head may bevisible in the image data captured by the driver-facing camera. In oneembodiment, a system in accordance with certain aspects of the presentdisclosure may be configured to determine a first keypoint 1301corresponding to the right eye of the driver and a second keypoint 1302corresponding to the left eye of the driver. The system may alsodetermine a third keypoint 1303 corresponding to the right ear of thedriver and a fourth keypoint 1305 corresponding to the left ear of thedriver. In one example, the “eye to ear vertical distance” 1305 will bethe distance between the 2 horizontal lines that is formed by a firstline (that connects the right eye keypoint 1301 and the left eyekeypoint 1302) and a second line (that connects the right ear keypoint1303 and the left ear keypoint 1304). This ‘eye to ear verticaldistance’ in one example may be used to detect the up and down movementof the head of the driver.

FIG. 14 shows the keypoints to calculate the vertical distance betweenthe eyes and ears in an embodiment that shows the above ‘eye to eardistance’ decreasing as the driver looks down. In this example, thesystem determines the ‘eye to ear vertical distance” 1405 is thedistance between the 2 horizontal lines that is formed by a first line(that connects the right eye keypoint 1401 and the left eye keypoint1402) and a second line (that connects the right ear keypoint 1403 andthe left ear keypoint 1404). This ‘eye to ear vertical distance’decreases from the perspective of the driver facing camera, as thedriver looks down. In one embodiment, this method may be one way todetect down and up movement of the head of the driver. In certainaspects of the present disclosure, this head movement may be correlatedto other events to detect and warn on anomalies.

FIG. 15 shows the keypoints of a driver looking straight ahead, with thedriver facing camera mounted in a specific position in the vehicle. Inone embodiment, the camera is mounted on the top left side of the drivernear the front windscreen of the vehicle. In this illustration, thedriver's head may be visible in the image data captured by thedriver-facing camera. In one embodiment, capturing the keypoints of theface and upper body of the driver, as seen from a specific mountposition of the camera, helps provide the natural posture and gaze ofthe driver as he is looking straight forward from the vehicle. In oneembodiment, the system determines the ‘eye angle’ 1505 that is the anglebetween the 2 lines that is formed by a first line (that connects theright eye keypoint 1501 and the driver facing camera 1506) and a secondline (that connects the left eye keypoint 1502 and the driver facingcamera 1506). In one example, this ‘eye angle’ is determined and notedby the compute device, for a driver that is looking straight ahead. Thesystem also determines the ‘eye to ear angle’ 1504 as the angle betweenthe 2 lines that is formed by a first line (that connects the left eyekeypoint 1502 and the driver facing camera 1506) and a second line (thatconnects the left ear keypoint 1503 and the driver facing camera 1506).The ‘eye angle’ and ‘eye to ear angle’ values, in one embodiment areused by the compute device to determine the normal values that indicatethe driver is looking straight ahead. Furthermore, as described above,by sampling multiple images in a preconfigured time interval, a median‘eye angle’ and ‘eye to ear angle’ angles may be determined that may beused as reference values when the compute device is trying to detectchanges to the pose and gaze of the driver.

FIG. 16 show the keypoints of a driver looking towards his left, withthe driver facing camera mounted in a specific position in the vehicle.In one embodiment, the camera is mounted on the top left side of thedriver near the front windscreen of the vehicle. In this illustration,the driver's head may be visible in the image data captured by thedriver-facing camera. In one embodiment, the system determines the ‘eyeangle’ 1604 that is the angle between the 2 lines that is formed by afirst line (that connects the right eye keypoint 1601 and the driverfacing camera 1606) and a second line (that connects the left eyekeypoint 1602 and the driver facing camera 1606). The system may alsodetermine in one example the ‘eye to ear angle’ 1605 that is the anglebetween the 2 lines that is formed by a first line (that connects theright eye keypoint 1601 and the driver facing camera 1606) and a secondline (that connects the right ear keypoint 1603 and the driver facingcamera 1606). These angles ‘eye angle’ and ‘eye to ear angle’ arecomputed and saved by sampling multiple images in a preconfigured timeinterval using the images captured by the driver facing camera 1606. Inone embodiment an increase in the value of these angles indicate thatthe driver is looking towards his left.

In some embodiments, the compute device may calculate pose as amultitude of distance and angle values between various keypoints, asdiscussed above. The current pose may be calculated at every sampleinterval of time and when the various keypoint distances and values arecompared to the “effective distance or effective angle values”, of theprevious preconfigured time interval, the compute device may determinethe current pose. The collection of various keypoints that determine thecurrent pose may be given a reference name, like, leaning forward,leaning backward, slouching down, back slumped down indicating sleepingor drowsy pose.

Similarly the current gaze and movement in gaze may now be calculated atevery sample interval of time. When these various distance and anglevalues between keypoints are compared to the “effective distance oreffective angle” of the previous preconfigured time interval, thecompute device may detect change of the direction of gaze of the driver.

In one embodiment, the compute device may use the above calculatedmedian values “effective distance or effective angle values”, andcorroborate this data with other data that is being retrieved by onboardsensors and other cameras. In one example, if the driver gaze is beingcalculated to be looking left, and the vehicle is turning left, asdetermined by road facing cameras, then there is no inattentiveness andthe driver need not be alerted for this detection of gaze movement.

To further make the gaze detection more accurate, multiple measurementsmay be checked against each other. In one embodiment, the direction thehead is turning may be detected by the compute device reading multiplekeypoint values. In one embodiment, if the Left Eye to the Left Eardistance reduces from its “effective distance” as calculated in the lastsample, then the compute device may detect this as a head movementtowards the left side. This left movement of the head may also bechecked by the monitoring the distance between the keypoints of Left andRight Eyes, i.e. ‘eye distance’. In one embodiment, if the ‘eyedistance’ distance increases, then the Head may be turning towards thecamera and if the camera is positioned on the left of the driver, itindicates that the driver's head is turning left. Thus, the computedevice may look at multiple keypoints to come to a conclusion that themovement of the head is in a certain direction. Multiple keypoint datahelps the compute device to increase confidence on the head movement andgive a more accurate gaze direction.

The shoulder distance increasing in value indicates that the driver isleaning forward, and may or may not be anomaly depending on the otherdata that is being read by onboard sensors and other cameras. Forexample, if the vehicle breaks are being applied, there will be a slightmovement of the driver towards the steering wheel given the laws ofmotion, and an increase of shoulder distance detection at the computedevice will not be an anomaly, and will not cause an alarm, since theother sensor readings will indicate to the compute device a vehiclebreaking condition.

In one embodiment, for gaze detection, the compute device may do animage processing of other well-known Landmarks on the face, which may bea multitude of points on the face. These other Facial Landmarks may becertain keypoints points on the face which have an impact on subsequenttask focused on the face, such as gaze detection. The Facial Landmarksmay be nose tip, corners of the eyes, chin, mouth corners, eyebrow arcs,ear lobes etc. The keypoints gathered by image processing of distancesand angles of the various landmarks on the face of the driver, will beable to give a more accurate picture of both the gaze and pose of thedriver.

Once the compute data has all the data from the above, it may generate adriver profile and save this data with the type of the vehicle, and thedriver profile, in a database on a cloud server to be used for laterusage. The data so collected may then be normalized in accordance withcertain aspects of the present disclosure. Accordingly, the normalizeddata may account for the position of the mounting of the camera, thevehicle type in which the camera was mounted, and the like.

Inward/Outward Alert Enhancement

Intelligent in-cab warnings may help prevent or reduce vehicularaccidents. In-cab warnings of unsafe events before or during the trafficevent may enable the driver to take action to avoid an accident. In-cabwarnings shortly after unsafe events have occurred may enable the driverto self-coach and learn from the event and how to avoid similar eventsin the future.

Industry standard ADAS in-cab alerts based on the outward environmentinclude forward collision warnings (FCW) and lane departure warnings(LDW). In-cab alerts based on the inward environment include drowsydriving. An NTSB study found that many drivers disable currentstate-of-the-art LDW systems due to too many unhelpful alerts.

First, current alerts may “cry wolf” too often when they are not needed,and cause drivers to ignore or turn-off the alerts reducing or removingtheir effectiveness. Second, are unsafe driving situations not currentlyhandled. Certain aspects of the present disclosure provide novelapproaches to addressing such issues.

In a first is a series of embodiments, inward and outward determinationsmay be combined to improve in-cab alerts. Accordingly, unnecessaryalerts may be reduced, and consequently more alerts may feel actionableto the driver leading the driver to respond to the alerts moreattentively and to keep the alerts active. According to certain aspectsof the present disclosure, an earlier warning may be provided if thedriver is distracted or determined to not be observing what ishappening.

According to certain aspects, a Forward Collision Warning (FCW) Alertmay be enhanced by taking into account a determination of the driver'slevel of distraction. In current state-of-the-art systems, an FCW alertmay be given if the time to collision with a vehicle in front dropsbelow a threshold value based on the relative speed of the vehicles.According to certain aspects of the present disclosure, and FCW may beenhanced. In one embodiment it may be determined if the driver iscurrently looking forward or not, and based on that determination adjustthe threshold time to collision before sounding the alert. For example,if the driver is looking in a direction other than forward, then if thetime to collision is 2.1 sec a FCW is sounded. If the driver is lookingforward likely seeing the vehicle, then the alert threshold may be 1.6seconds. This affords the driver more time to respond when alreadyobserving what is happening and reduces the number of “crying wolf”alerts that are just alerting the driver to what they are alreadyobserving.

In an alternative embodiment of certain aspects of the presentdisclosure, an FCW threshold may be kept at 2.1sec when the driver isdetermined to be looking forward, and increased to 2.6sec when thedriver is looking elsewhere or determined to be distracted, to give thedriver more time to react as he/she needs to look forward and understandthe scene.

In the base-line of this feature the driver may be determined to bedistracted based sole on determining the drivers gaze or head directionto be looking forward. A further enhancement may include determining ifthe angle of the driver's gaze is in the direction of the object ofinterest to determine if the driver may be perceiving that object. Thedriver's gaze may be determined using computer vision techniques.

A state-of-the-art Lane Departure Warning (LDW) may be triggered if avehicle leaves its lane of travel. This creates a lot of “cry wolf”events, as every lane change is alerted. The system may determine if aturn signal is on when the lane change occurs, so that only lane changesthat occur when the turn signal is off may be alerted. According tocertain aspects of the present disclosure, an inward camera may also beused to determine if the driver makes a shoulder check gaze in thedirection of lane change before changing lanes, and suppressing thealert if such a maneuver is made. This may reduce the number of intendedlane changes that trigger a “cry-wolf” alert sound.

Further, even if a driver signals a lane change, but is determined tonot have checked that the lane is clear before changing lanes, then acoaching alert may be made after the lane change. Gaze detection in theearlier section of ‘Gaze and Pose Detection’ would help correlate thedriver movements with movements the vehicle is making, like lanechanges, and if the gaze of the driver looked in that direction beforethe lane change.

In another embodiment there may be an adjustable threshold concerninghow long to wait while a driver looks away from the road before alertingthe driver to their distracted state. In one embodiment, the thresholdtime may be a function of the outward scene with two threshold times. Ifthe road ahead of the driver does not have any vehicles within a givendistance of travel time, and the driver is maintaining his/her laneposition, then the threshold time that the driver may look away from theroad before an alert is sounded may be set to the long threshold time.If there are vehicles detected in the road ahead or the lane positionvaries by more than a set threshold, then the short threshold time maybe used.

In another embodiment, the use of a mobile phone by a driver may bemonitored by the driver facing camera with the gaze detection methods inthe previous section of ‘Gaze and Pose Detection’. A warning may beissued to the driver if the threshold time of the downward gaze towardsthe mobile phone is longer than a pre-configured safety period. Inanother embodiment, in case the outer looking camera is showing othervehicles close to this vehicle and if the speed of the vehicle is abovea certain threshold limit (e.g., 15 miles per hour), an alert messagemay be issued to the driver and logged in the compute device and remotecloud server.

Another embodiment may use a series of threshold times or a thresholdfunction that takes as inputs one or more of the distance to the nearestvehicle, number of vehicles on the road, lane position, vehicle speed,pedestrians present, road type, weather, time of day, and the like, todetermine a threshold time.

Many other alert thresholds are contemplated for which the threshold maybe varied for inward alerts based on the determination of the outwardscene complexity, and vice versa. That is, the threshold of an outwardalert may be based on a determination of the inward scene complexity. Inaddition, there are a number of additional unsafe driving events thatmay be captured and that may be a basis for issuing a warning alert tothe driver. Several examples are described in detail below.

Red Light Ahead

According to certain aspects of the present disclosure, a red-lightahead alert may be enhanced. In one embodiment, a vision system maydetect a traffic light in front of the vehicle, and may determine thetraffic light state as green, yellow, or red. A determination of thedistance to the traffic light is made in one of many ways, such as GPSdetermined location against a map, visual distance to the intersectionbased on size of objects in pixels and known object sizes, distance tothe intersection based on camera known intrinsics and extrinsics andintersection threshold compared to vanishing point, radar based distancemeasurements, stereo vision measurements, or other approaches, as wellas combinations of techniques and approaches. A determination of thevehicle speed may be made using GPS measurements, built-in vehicle speedindications based on wheel rotations, vision odometry, inertials, orother methods, or combinations of methods. In a base-line embodiment, ifthe time to the intersection determined based on the vehicle speed andintersection distance drops below a threshold value and the trafficlight state is red, then a red light ahead alert is sounded.

In another embodiment, the threshold value may be varied based on adetermination of driver distractedness and/or determination of scenecomplexity.

In still another embodiment, a traffic light state machine model may beused, either a location-agnostic model or a location-specific modelbased on the region around traffic lights and/or the specific trafficlight intersection. A simple model may predict an expected time fromwhen the light turns yellow until the light turns red. Based on thistime, then in this embodiment, even if the light is yellow, if it isdetermined that the light will turn red before the vehicle enters theintersection then a red light ahead alert may be triggered.

In some embodiments, the model may be used to determine a stale greenlight that would be used to estimate if the light would turn red beforethe vehicle would arrive.

In still another embodiment, rather than the time to the intersection,another function of the distance and/or speed may be used. For example,if the distance to the intersection is less than a threshold and thespeed is above a threshold, then a determination may be made.

In an additional embodiment, if the driver presses on the brake, whichmay be determined either by a CANBUS or vehicle indication that thebrake was pressed, by an inertial sensor based determination, or by aGPS speed measurement decreasing, or some combination, then the redlight ahead alert may be suppressed since this may be an indication thatthe driver is already aware of the red light ahead. In a variation ofthis embodiment, the threshold time to the red light may be reduced, andthe alert still triggered if the driver goes below that reducedthreshold.

In an additional embodiment, an outward camera or vehicle determinedposition applied to a map may be used to determine which lane thevehicle is traveling in, such as left turn, straight ahead, or rightturn lane. Or alternatively, the system or method may incorporate driverindications, such as turn signs, to then further determine the driversintended actions and further map the appropriate traffic light for thered-light ahead alert. For example, if the driver is traveling in theleft turn lane and the left turn lane turn arrow light is red while thestraight ahead light is green, and the time to intersection crossing maybe less than the threshold then a red light ahead alert may betriggered.

Light Turned Green

With the increase in driver distraction, there are increasingoccurrences where a driver is stopped at a red light, distracted andlooking away from the light, and does not notice that the light turnedgreen. This may increase the risk of a rear-end collision if a driverbehind doesn't realize the distracted driver hasn't started move despitethe green light. This may also cause frustrations for other driversbehind the driver and risk of road rage.

According to certain aspects of the present disclosure, a light-turnedgreen alert may let the driver know that the light has turned green. Ina baseline embodiment, a vehicle mounted camera looking forward maydetermine that the vehicle is stopped at an intersection with a redlight. When the visual detector detects that the light turns green analert is triggered for the driver.

Furthermore, in some embodiments, the time that the light is green andthe vehicle is not moving may be determined. If that time goes above athreshold, then an alert is triggered. This may reduce the frequency of“cry-wolf” alerts.

In another embodiment, an inward camera may determine if the driver isdistracted, and only trigger a light-turned-green alert if the driver isdetermined to not be looking forward. Further, if a threshold time fromthe green light is being used, then the driver distraction may be usedto determine the threshold time until a light-turned-green alert istriggered. Accordingly, a driver looking forward would have a longertime than a distracted driver before an alert is given.

In a further enhancement, a traffic light model may be used to estimatewhen the red light might turn green based on noting when the lightturned red, and potentially other determined features, such as thespecific intersection or general location statistics, vehicle movementsfor the cross traffic, and or turning traffic, among others. Then apre-alert may be triggered to the driver that the light is about to turngreen. This pre-alert may be modified or suppressed based on a driverdistracted determination.

In another embodiment, the outward camera or determined position appliedto a map may determine the lane or characteristic of the intersectionlane that the vehicle is in, such as a left turn lane, straight aheadlane, or right turn lane, and then maps and uses the appropriate trafficlights for determining the light-turned-green alert.

Train Tracks Ahead Alert or Stop Sign Ahead Alert

Many types of vehicles are required to stop at all train tracks, such asschool buses and vehicles carrying hazardous cargo. Additionally, allvehicles are required to stop at stop signs.

According to certain aspects of the present disclosure, a Train TrackAhead Alert or Stop Sign Ahead alerts the driver if configured to warnfor train tracks ahead and/or stop sign ahead. In a baseline version itmay alert whenever the driver approaches the intersection of traintracks or stop sign. The train tracks and/or stop sign may be determinedby visual detection from a camera of signs or features indicating theintersection or by mapping the vehicle position on to a map indicatingthe intersection.

Additional embodiments may be similar to the embodiments of the redlight ahead warning alert, such as measuring the vehicle speed anddistance to the intersection to determine a time to the intersection andsounding the alert if the time to the intersection goes below athreshold. Further, varying that threshold based on a determination ofdriver distractedness.

Audio Features for Alerts

In determining the inward scene complexity and/or outward scenecomplexity an audio sensor may be used to help determine the scenecomplexity.

In the above examples, distracted driving was used an example of abroader class of inward scene metrics, which may be referred to as aninward scene complexity. That complexity may include distractions due todriver looking in the cab, shoulder checking, eating food, talking onthe phone (hands free or in hands), playing with the radio, texting,talking to other passengers, being drowsy, sleeping, handling children,among other elements.

An audio sensor such as one or more microphones may be used to helpdetermine both inward and outward scene complexity. For example, thesound of cars screeching, sirens from emergency vehicles, and honkingmay indicate different levels of outward scene complexity. Similarly,sounds of the radio playing, conversations, driver talking, babiescrying, among others may indicate different levels of inward scenecomplexity. In an embodiment a classification of an audio signal may beused to determine the presence of each of these or other events. Thenthe scene complexity indication function may take these into account andtherefore impact the thresholds.

In one embodiment, if the driver is detected as talking with apassenger, then a higher cognitive load may be assumed for the driverand an assumed slower reaction time, so a the FCW may have a higherthreshold to give an earlier warning.

In an embodiment for the light turned green warning, if a car horn isdetected then it may be assumed that the driver has heard an impliedexternal alert of that horn, so a higher threshold for a longer durationof green may be used before alerting the driver.

In another embodiment, if a siren is heard, then the light-is-greenwarning may be suppressed so as not to accidentally encourage the driverto disrupt an emergency vehicle.

Enhancements of Internally Focused Alerts with External or IMU Inputs

According to certain aspects of the present disclosure, internallyfocused alerts such as distracted or drowsy detection may be enhancedbased on Inertial Measurement Unit input and/or outward facing camerainput. One embodiment may include confirming or changing drowsythresholds based on decreasing speed (drifting off to sleep and slowingdown). A second embodiment may include confirming or changing drowsythresholds based on lack of steering input (lack of lateral IMU)steering correction prior to lane drift. A third embodiment may includeconfirming or changing drowsy thresholds based on lack of other vehicleson the road/low ambient light/trip length (time or ratio) since vehiclehas “seen” another vehicle in frame.

Additional Embodiments

At least some aspects of the present disclosure will now be describedwith reference to the following numbered clauses:

-   Clause 1. A method, comprising: determining an indication of an    inward driving scene complexity; adjusting at least one safety    threshold based on the determined indication; and determining a    potentially unsafe driving maneuver or situation based on that at    least one safety threshold.-   Clause 2. The method of clause 1, wherein the indication is based on    a level and/or type of distracted driving behavior.-   Clause 3. The method of clause 1, wherein the safety threshold    corresponds to an outward driving scene.-   Clause 4. A method comprising: determining an indication of an    outward driving scene complexity; adjusting at least one safety    threshold based on the determined indication; and determining a    potentially unsafe driving maneuver or situation based on that at    least one safety threshold.-   Clause 5. The method of clause 4, wherein the safety threshold    corresponds to an inward driving scene.-   Clause 6. The method of clause 1 or clause 4, wherein the    potentially unsafe driving maneuver comprises at least one of a    forward collision warning, a lane departure warning, a red-light    ahead warning, a railroad crossing ahead warning, a stop sign ahead    warning, and a driver distracted warning.-   Clause 7. A method, comprising: determining an indication of an    inward driving scene; adjusting at least one safety threshold based    on the indication; and detecting an occurrence of a driving event    based at least in part on the adjusted at least one safety    threshold.-   Clause 8. The method of clause 7, further comprising: transmitting    event data to a remote server based on the determined occurrence of    the driving event.-   Clause 9. The method of clause 7, further comprising: triggering an    audible or visual alert based on the determined occurrence of the    driving event.-   Clause 10. The method of clause 7, wherein the indication of an    inward driving scene corresponds to at least one of: a gaze    direction of a driver, a presence of a communication device, an    absence of a fastened seatbelt, or a frequency of yawning, talking,    or blinking.-   Clause 11. The method of clause 7, wherein the safety threshold    corresponds to at least one of a speed; a distance from a traffic    light; a distance from an intersection; a distance from a stop sign;    a distance to a railroad crossing; a lateral velocity; a distance    from a lane or a road boundary; a lateral distance from a vehicle in    an adjacent lane; a forward distance to an obstacle; or a speed of    the obstacle.-   Clause 12. The method of clause 7, wherein the driving event is at    least one of a traffic-light crossing, a stop sign crossing, a    railroad crossing, a lane departure, or a potentially unsafe    following distance.-   Clause 13. The method of clause 7, wherein the inward driving scene    corresponds to a driver performing a visual check in the direction    of a lane change; the driving event is a potentially unsafe lane    departure; and the safety threshold is adjusted so that the    detection of a potentially unsafe lane departure event is    suppressed.-   Clause 14. A method comprising: determining an indication of an    outward driving scene; adjusting at least one safety threshold based    on the indication; and determining an occurrence of a driving event    based at least in part on the adjusted at least one safety    threshold.-   Clause 15. The method of clause 14, further comprising: transmitting    event data to a remote server based on the determined occurrence of    the driving event.-   Clause 16. The method of clause 14, further comprising: triggering    an audible or visual alert based on the determined occurrence of the    driving event.-   Clause 17. The method of clause 14, wherein the indicated outward    driving scene corresponds to at least one of: a location of a    traffic light, an illuminated color of the traffic the traffic    light, an elapsed time since a change in the illuminated color of    the traffic light, or a driver's speed.-   Clause 18. The method of clause 14, wherein the indication of an    outward driving scene corresponds to at least one of: a lateral    velocity; a distance from a lane or a road boundary; or a lateral    distance from a vehicle in an adjacent lane.-   Clause 19. The method of clause 18, wherein the safety threshold    corresponds to at least one of: a gaze direction of a driver or a    duration of a deviated gaze direction.-   Clause 20. The method of clause 14, wherein the driving event    corresponds to at least one of distracted driving, drowsy driving, a    driver failure to check a blind-spot prior to a lane change, a    driver performing a visual check in the direction of a lane change.-   Clause 21. A method of calibrating a driver-facing camera,    comprising: receiving image data from a camera coupled to a vehicle;    locating a plurality of keypoints in the image data, wherein a first    keypoint of the plurality of keypoints corresponds to a location on    a driver of the vehicle; determining a distance between the first    keypoint and a second keypoint from the plurality of keypoints;    determining a speed of the vehicle; and updating an estimated    typical distance between the first keypoint and the second keypoint    based on the determined distance, when the determined speed is above    a predetermined threshold.-   Clause 22. The method of clause 21, wherein the first keypoint    corresponds to a location in the image data corresponding to a first    shoulder of the driver, and the second keypoint corresponds to a    location in the image data corresponding to a second shoulder of the    driver.-   Clause 23. The method of clause 21, further comprising: receiving a    second image data from the camera; determining a second distance    between a pair of keypoints in the second image data, wherein the    pair of keypoints in the second image data corresponds to the first    keypoint and the second keypoint in the image data; determining a    scaling factor based on the determined second distance and the    estimated typical distance.-   Clause 24. The method of clause 23, further comprising: determining    a third distance between a third pair of keypoints; and determining    a deviation from a typical pose based at least in part on the third    distance and the scaling factor.-   Clause 25. The method of clause 24, wherein the third pair of    keypoints corresponds to two eyes of the driver; and further    comprising: determining a gaze direction of the driver based on the    distance between the two eyes in the second image data, and    estimated typical distance between the two eyes, and the scaling    factor.-   Clause 26. The method of clause 25, wherein the determined gaze    direction is further based on a distance between a detected ear of    the driver and a detected nose of the driver.-   Clause 27. The method of clause 24, further comprising: alerting the    driver based on whether the determined deviation exceeds a    predetermined threshold.-   Clause 28. The method of clause 27, wherein alerting the driver    comprises audio feedback to the driver.-   Clause 29. The method of clause 27, wherein alerting the driver    comprising transmitting alert data to a remote server, so that the    driver or an agent of the driver may review the alert data.-   Clause 30. The method of clause 27, further comprising: receiving    outward image data from a road-facing camera coupled to the vehicle;    determining a driving behavior based at least in part on the outward    image data; and wherein alerting the driver is further based on the    determined driving behavior.-   Clause 31. The method of clause 21, wherein the deviation from the    typical pose corresponds to the driver looking in the direction of a    lane change, and wherein the driving behavior is the lane change.-   Clause 32. The method of clause 21, wherein updating an estimated    typical distance comprises determining a median of determined    distances between keypoints corresponding to the first keypoint and    the second keypoint.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The processing system may be configured as a general-purpose processingsystem with one or more microprocessors providing the processorfunctionality and external memory providing at least a portion of themachine-readable media, all linked together with other supportingcircuitry through an external bus architecture. Alternatively, theprocessing system may comprise one or more specialized processors forimplementing the neural networks, for example, as well as for otherprocessing systems described herein.

Thus, certain aspects may comprise a computer program product forperforming the operations presented herein. For example, such a computerprogram product may comprise a computer-readable medium havinginstructions stored (and/or encoded) thereon, the instructions beingexecutable by one or more processors to perform the operations describedherein. For certain aspects, the computer program product may includepackaging material.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein may bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device may be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein may beprovided via storage means (e.g., RAM, ROM, a physical storage mediumsuch as a thumb drive, etc.), such that a user terminal and/or basestation may obtain the various methods upon coupling or providing thestorage means to the device. Moreover, any other suitable technique forproviding the methods and techniques described herein to a device may beutilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes and variations may be made in the arrangement, operation anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A method comprising: capturing, by at least oneprocessor of a computing device with an outward facing camera on or in avehicle, visual data outside of the vehicle; detecting, by the at leastone processor based on the visual data, a red light turned to a greenlight for the vehicle; and responsive to detecting, by the at least oneprocessor, that the vehicle has not moved for at least a thresholdperiod of time since detecting the red light turned to the green light,generating, by the at least one processor, an alert for a driver of thevehicle.
 2. The method of claim 1, further comprising responsive todetecting, by the at least one processor, a sound of a horn afterdetecting the red light turned to the green light, extending, by the atleast one processor, the threshold period of time.
 3. The method ofclaim 1, further comprising responsive to detecting, by the at least oneprocessor, a sound of a siren after detecting the red light turned tothe green light, suppressing, by the at least one processor, the alertto the driver of the vehicle.
 4. A method comprising: capturing, by atleast one processor of a computing device with an outward facing cameraon or in a vehicle, visual data outside of the vehicle; detecting, bythe at least one processor based on the visual data, a red light turnedto a green light for the vehicle; and responsive to detecting, by the atleast one processor, that a driver of the vehicle is not looking at thegreen light, generating, by the at least one processor, an alert for adriver of the vehicle.
 5. The method of claim 4, wherein detecting thatthe driver of the vehicle is not looking at the green light comprisesdetecting that the driver is looking into the cab of the vehicle,shoulder checking, eating, interacting with a dashboard of the vehicle,using a phone, gazing at a phone, texting, talking to a passenger of thevehicle, sleeping, appearing drowsy, or handling children.
 6. The methodof claim 4, wherein detecting that the driver of the vehicle is notlooking at the green light comprises detecting, using a microphoneinside of the vehicle, sound inside of the vehicle.
 7. The method ofclaim 4, wherein detecting that the driver of the vehicle is not lookingat the green light comprises detecting, using a microphone outside ofthe vehicle, sound outside of the vehicle.
 8. The method of claim 7,wherein sound outside of the vehicle comprises a siren, a carscreeching, or a horn honking.
 9. The method of claim 8, furthercomprising responsive to detecting, by the at least one processor, thehorn honking after detecting the red light turned to the green light,transmitting, by the at least one processor, the alert after a period ofdelay.
 10. The method of claim 1, further comprising responsive todetecting, by the at least one processor, the siren after detecting thered light turned to the green light, suppressing, by the at least oneprocessor, the alert to the driver of the vehicle.
 11. A methodcomprising: capturing, by at least one processor of a computing devicewith an outward facing camera on or in a vehicle, visual data outside ofthe vehicle; detecting, by the at least one processor based on thevisual data, a traffic light corresponding to a lane of the vehicle;detecting, by the at least one processor based on the visual data, a redlight turned to a green light for the traffic light corresponding to thelane of the vehicle; and generating, by the at least one processor, analert for a driver of the vehicle when the traffic light correspondingto the lane of the vehicle turned to the green light.
 12. The method ofclaim 11, further comprising: detecting, by the at least one processorbased on the visual data, a second traffic light corresponding to anadjacent lane of the vehicle; detecting, by the at least one processorbased on the visual data, a second red light turned to a second greenlight for the second traffic light corresponding to the adjacent lane;and suppressing, by the at least one processor, the alert for the driverof the vehicle when the second traffic light corresponding to theadjacent lane of the vehicle turned to the green light.
 13. The methodof claim 11, further comprising responsive to detecting, by the at leastone processor, that the vehicle has not moved for at least a thresholdperiod of time since detecting the red light turned to the green light,transmitting, by the at least one processor, an alert to a driver of thevehicle.
 14. The method of claim 13, further comprising responsive todetecting, by the at least one processor, a sound of a horn afterdetecting the red light turned to the green light, extending, by the atleast one processor, the threshold period of time.
 15. The method ofclaim 13, further comprising responsive to detecting, by the at leastone processor, a sound of a siren after detecting the red light turnedto the green light, suppressing, by the at least one processor, thealert to the driver of the vehicle.
 16. The method of claim 11, furthercomprising detecting, by the at least one processor, that the driver ofthe vehicle is not looking at the green light.
 17. The method of claim16, wherein detecting that the driver of the vehicle is not looking atthe green light comprises detecting that the driver is looking into thecab of the vehicle, shoulder checking, eating, interacting with adashboard of the vehicle, using a phone, gazing at a phone, texting,talking to a passenger of the vehicle, sleeping, appearing drowsy, orhandling children.
 18. The method of claim 16, wherein detecting thatthe driver of the vehicle is not looking at the green light comprisesdetecting, using a microphone inside of the vehicle, sound inside of thevehicle.
 19. The method of claim 16, wherein detecting that the driverof the vehicle is not looking at the green light comprises detecting,using a microphone outside of the vehicle, sound outside of the vehicle.20. The method of claim 19, wherein sound outside of the vehiclecomprises a siren, a car screeching, or a horn honking.